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Abstract 

Program assessment has been a topic of significance in higher education 
over the last decade. Initially, program assessment was implemented in 
response to concerns over the quality of college graduates’ education 
and increasing competiveness in the job market for American college 
graduates. Recently, higher education institutions have been encouraged to 
engage in assessment which can generate data that is comparable between 
institutions; resulting in assessment models which are predominantly 
quantitative. While quantitative data aids in cross-institution comparison, 
which remains important, alone it lacks the specificity that is needed to 
understand nuanced experiences of students, faculty, and administration. 
In this article, the assessment model that has been developed and 
implemented at one small-sized private institution is described. Focus 
is given to the qualitative elements of the multiple-method model. An 
evaluation of the model, post-implementation, is included in order to 
provide detail to facilitate potential application at other institutions. 
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.institutional assessment has become a topic of cumulative importance, and 
even great debate, amongst higher education professionals, and within higher education 
institutions, over the last two decades (Halpern, 2013). Both regional and secondary 
accreditation organizations have imposed rigorous assessment requirements and higher 
education institutions have become accustomed to environments that are characterized 
by assessment and oversight. Regional accrediting organizations provide guidelines and 
expectations for accreditation; however, the design and implementation of assessment 
procedures are at the discretion of the higher education institution (e.g., Northwest 
Commission on Colleges and Universities, 2013). Thus, higher education institutions 
must develop a formal assessment methodology that meets accountability standards of 
accrediting organizations. Yet, a lack of consensus and specificity regarding the ways in 
which assessment ought to be conducted within an institution remains. Though quantitative 
assessment methods have historically been preferred within higher education, a shift in 
the field of higher education has emerged as educational researchers have come to realize 
that important information and questions cannot necessarily be gathered or answered 
by a single methodology (Commander & Ward, 2009; Van Note Chism & Banta, 2007). 
Furthermore, Howard and Borland (2001) noted that institutional assessment is actually a 
form of qualitative research since institutional research is context-focused and unique to 
the specific university. Considering this information, qualitative research methodologies 
and frameworks are important elements that must be included within institutional research. 
This does not exclude or nullify the importance of quantitative measures and methods; 
rather this information highlights the importance of integrating both qualitative and 
quantitative inquiry into institutional assessment (Howard & Borland, 2001). Therefore, 
integrating qualitative methods and frameworks into institutional assessment practices 
increases access to important data. 
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A Brief History of Higher Education Assessment 

Traditionally, accountability in higher education meant that institutions used financial 
resources effectively and appropriately (Ewell & Jones, 2006; Liu, 2011). However, at the turn 
of the century, a shift in the concept of higher education accountability began to emerge. The 
new form of accountability mandated institutions provide evidence that they were producing 
a qualified workforce through assessing learning and performance outcomes (Contreras- 
McGavin & Kezar, 2007; Ewell & Jones, 2006; U.S. Department of Education, 2006). The 
purpose of these learning and performance outcomes was to measure students’ mastery of 
knowledge and skills, and they have become important and contentious topics within the 
field of higher education (Contreras-McGavin & Kezar, 2007; Furman, 2013; Ilalpern, 2013). 
This change in accountability appears to be in response to several factors (U.S. Department 
of Education, 2006). As the quality of higher education in other countries began to surpass 
the United States, it became apparent that the United States’ higher education system was 
falling behind. In addition, employers began to criticize higher education institutions for 
failing to produce a qualified workforce with basic writing, problem-solving, critical thinking, 
and leadership skills (Liu, 2011; U.S. Department of Education, 2006). Thus, the demand for 
qualified graduates and effective educational programs has resulted in an emphasis in higher 
education on learning and performance outcomes. As a result, discussion in the literature 
about creating universal, or at least, comparable across institutions, outcome assessment 
processes has emerged (Contreras-McGavin & Kezar, 2007; Hanson & Mohn, 2011; Liu, 2011). 
In addition, ideas about how to measure student learning and report these outcomes have been 
debated (Furman, 2013; Germaine, Barton, & Bustillos, 2013). 

Outcomes Assessment 

Institutional comparison. The recommendation that institutions measure their 
performance outcomes in such a way as to be easily comparable to other institutions does not 
account for the various roles and unique missions of different institutions within the higher 
education landscape (Ewell, 2013; Ewell & Jones, 2006; Liu, 2011). Liu (2011) questioned if 
it is meaningful or helpful to compare performance outcomes of premier research universities 
with liberal arts baccalaureate colleges. Instead, Liu suggested that a more helpful comparison 
would be among similar-type institutions. Furthermore, performance and learning outcomes 
can be useful tools in assessing an institution’s effectiveness in fulfilling their distinct role 
and purpose (Ewell & Jones, 2006; Germaine et al., 2013). Thus, performance and learning 
outcomes are beneficial for evaluating an institution’s ability to fulfill its mission in comparison 
with similar-type institutions. 

Methods for measuring outcomes. Measuring certain outcomes and unique roles 
of higher education institutions can be difficult; some outcomes (e.g., leadership, self- 
awareness, critical thinking, mission fulfillment, etc.) are not easily or efficiently measured 
quantitatively (Contreras-McGavin & Kezar, 2007; Furman, 2013; Germaine et al., 2013). 
Though these outcomes are important goals of higher education, quantitative measures for 
these outcomes typically involve lengthy self-reported surveys (Furman, 2013). Furthermore, 
the desire for an institutional-comparable assessment process encourages the development of a 
simplistic, broad quantitative approach that can be applicable to multiple types of institutions 
(Contreras-McGavin & Kezar, 2007). However, the demand for detailed information about 
program effectiveness, student learning, and fulfillment of unique roles on difficult to measure 
outcomes is more attainable through the use of qualitative methods (Contreras-McGavin & 
Kezar, 2007; Harper & Kuh, 2007; Van Note Chism & Banta, 2007). Thus, there is a paradox 
within the literature on how to assess outcomes: quantitatively and institutionally comparable 
or qualitatively with an emphasis on the institutions’ role and purpose. 

Outcomes assessment process. Despite the increasing emphasis on learning and 
performance outcomes, there is very little information or recommendations for developing a 
formal assessment process (Contreras-McGavin & Kezar, 2007). However, a close connection 
between the formal assessment process, especially the evaluation of learning outcomes, and 
the institution’s overall mission and purpose is recommended (Contreras-McGavin & Kezar, 
2007; Germaine et al., 2013; Ilalpern, 2013). Considering this literature, we believe that it 
is essential to include qualitative methods within institutional assessment. Specifically, 
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qualitative methods are useful for evaluating in-depth, specific outcomes for less quantifiable 
educational goals and allow institutions to evaluate their effectiveness in fulfilling their unique 
role and mission. 

In recent years, there has been an increase in literature published about the possible 
uses of qualitative methods in institutional assessment (e.g., Contreras-McGavin & Kezar, 
2007; Fifolt, 2013; Harper & Kuh, 2007; Museus, 2007; den Outer, Handley, & Price, 2013; 
Van Note Chism & Banta, 2007), however, very little has been published on the applied use of 
qualitative methods in a formal assessment process. Furthermore, there are few publications 
on higher education institutions’ formal assessment processes. Therefore, the purpose of this 
article is to expand the literature on (a) institutional assessment processes by presenting one 
university’s assessment procedures, and (b) the applied use of a qualitative framework for 
institutional research. 


Institutional Profile 


Methodology 
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The university is a private, regionally-accredited, Christian co-educational liberal 
arts institution founded in 1934. The university maintains campuses in multiple states and 
numerous extension sites; in addition, the offering of online courses, degrees, and programs 
has rapidly expanded for traditional, adult, and graduate students. The university employs 
five senior administrators, 69 full-time faculty members, and a pool of 311 adjunct faculty 
members to award associate, baccalaureate, master, and doctoral degrees. 

The total fall 2013 enrollment was approximately 1,740 students. On-campus residence 
is encouraged for traditional undergraduates and 674 of the 958 undergraduate students live in 
on-campus residence halls and student apartments. Students participate in a wide range of co- 
curricular activities, including student government, residence life, campus ministries, student 
organizations, intramural athletics, and intercollegiate athletics for men and women. 


Applied Framework/Instrument Design 

In response to revisions and new requirements in the regional accreditation process, 
the university’s Accreditation Committee, composed of a mix of administrators, faculty, 
and staff, created an institution-wide assessment process. The purpose of the assessment 
process was to be sufficiently broad so as to effectively function at both the institutional 
and departmental level: (a) to measure institutional mission-fulfillment and, (b) to facilitate 
departmental accountability and improvement. In addition, the process was to be sufficiently 
flexible so as to encompass both quantitative and qualitative departmental assessment. 


Assessment matrix. Central to the assessment process was the Assessment Matrix 
(Appendix A). In order to assess both the overall mission fulfillment and the work of individual 
departments, an Assessment Matrix was established that was applicable to all areas of the 
institution, from academic departments and student development to administrative support 
offices and facilities maintenance. Along with outcomes, indicators, findings, and an analysis 
of findings, the matrix delineated a standard for success and required that a success score is 
assigned to every outcome. Through the use of this Assessment Matrix, the university was able 
to gain a summative evaluation of departmental success and institutional mission fulfillment. 

Assessing mission fulfillment. In accordance with the new regional accreditor 
requirements, the mission of the university was detailed in four core themes (community, 
spirituality, academics, and engagement). The university defined institutional mission 
fulfillment in relation to the success of core theme outcome achievement measured using the 
Assessment Matrix for each core theme. 
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These core themes were specifically created to be sufficiently broad in order to 
be applicable to all university functions. Therefore, in the completion of a departmental 
Assessment Matrix, the department director selected the most appropriate core theme that 
is addressed by each departmental outcome. For example, while an academic major outcome 
might be assigned the academic core theme, the security office might perceive their outcome 
that encompasses their work toward student safety as addressing the community theme. 
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The matrices from all areas of the university were then compiled by core theme and 
core theme outcome to facilitate institution-wide assessment of mission-fulfillment through 
core theme achievement. Specifically, the university defined institutional mission fulfillment 
as an overall average success score on the core themes of 3.0 or higher on a 4.0 scale. Thus, 
every department on campus contributed to institutional mission fulfillment through the 
achievement of their departmental outcomes and associated core themes. 

Assessing departmental accountability and improvement. In addition to measuring 
mission-fulfillment, the Assessment Matrix provided a structured process for facilitating 
accountability and improvement. Every department on campus engaged in ongoing systemic 
assessment through the use of the Assessment Matrix. The staff department assessment 
matrices assessed performance-based outcomes, measuring the primary departmental 
responsibilities, tasks, and functions. In contrast, the academic department assessment 
matrices were primarily learning-based outcomes, measuring the primary student learning 
goals within a major. All outcomes were framed in terms of a customer, who may be a student, 
parent, donor, alumnus, or faculty and/or staff in another department. The utilization of both 
qualitative and quantitative means of outcome (and indicator) assessment were encouraged 
in order to most appropriately, effectively, and comprehensively measure and interpret the 
data in terms of departmental performance and/or student learning. This assessment occurred 
throughout the year, with staff following the Assessment Strategy Cycle (Appendix B) and 
academic departments following the Academic Assessment Cycle (Appendix C). 

Assessment cycle. Although variation between the staff and academic cycles existed, a 
similar timeline was followed by all departments. Prior to the commencement of the traditional 
academic year, the outcomes, indicators, data source, time period, and standards for success 
were established and core themes assigned. The collection of data occurred throughout the 
year. Upon the completion of the academic year, the data were analyzed for findings and based 
on the results, a success score assigned. Finally, based on this analysis, subsequent program 
changes were planned (e.g., added or deleted outcomes and/or indicators, adjusted standard 
for success, implemented program changes for next assessment cycle, etc.). Upon completion 
of the Assessment Matrix, the finalized document was sent to the respective vice president and 
the Office of the Provost. The cycle then began again for the subsequent year with a matrix 
that reflected the changes made. 

Initially, as part of departmental planning, the vice president of the appropriate 
division reviewed and provided feedback on the department outcomes, indicators, and 
standard for success. Upon completion of the assessment cycle the vice president of the 
appropriate division reviewed and evaluated the subsequent findings. This involvement 
fostered shared expectations and increased communication between the vice president and 
department directors. The faculty, specifically the program directors and the deans, evaluated 
the academic programs each year and implemented changes for improvement, including 
curriculum changes. This assessment was based on student achievement as documented 
through the student’s completion of the learning outcomes for each course, major, and degree 
as indicated by the instructors and program directors. 

Thus, the findings of the assessment matrices were then used to determine what, if 
any, changes were needed in the departmental outcomes, indicators, and/or success scores 
in order to most accurately assess the work of the staff department or the learning of the 
students within a major. This process ensured that assessment is reviewed at all levels of 
the institution, with the Assessment Matrix findings included in divisional reports to the 
Board of Directors, yet empowered the staff and faculty to lead the changes necessary for 
departmental improvement. 

Procedures 

By utilizing the process described above, the university established a system of 
assessment that highlighted both worthy achievement and areas necessitating improvement. 
In addition, this model encouraged investment in and growth toward increased mission 
fulfillment. The flexibility of this process to include both qualitative and quantitative 
assessment was a distinctive strength of this applied assessment approach. In particular, 
the ability of both staff and academic departments to utilize qualitative assessment methods 
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to measure their outcomes and indicators provided a more nuanced understanding of the 
individual experience and better protected and communicated the voices of those involved in 
the outcome. In addition, the establishment of standards for success rubrics by department 
directors was qualitative in nature, based on their analysis of observations, interviews, and past 
experience. For example, for student advising outcomes within the X major, the department 
chair created the standard for success rubric based on individual student interviews, class 
focus group feedback, and past experience in supporting struggling students. Thus, this process 
intentionally provided space for the diversity of stories and experiences of the students and 
customers served by the various departments. 
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Standards for success. The Standard for Success Rubrics undergird the Assessment 
Matrix as this stage includes both the definition and measurement of departmental success. 
The development of the Standard for Success Rubrics was intentionally qualitative, as the 
detailed information about program effectiveness and student learning on difficult to measure 
outcomes was most effectively obtained through the use of qualitative methods (Contreras- 
McGavin & Kezar, 2007; Harper & Kuh, 2007; Van Note Ghisnr & Banta, 2007). 

The Success Score Rubric was described as follows: 

4 = significantly exceeded the Standard for Success 

In the Success Score Rubric, a 4 signifies that the department or program has fully reached 
its goals in this area. This score indicates that no improvement or change is necessitated 
and maintaining current practice will ensure the ongoing success of the department and its 
contribution to the continued health and wellbeing of the University. 


3 = met the Standard for Success 


In the Success Score Rubric, a 3 signifies that room for minor improvement remains and/ 
or is possible for the specific departmental goal, but that at the present time, this item has 
been sufficiently fulfilled for the ongoing success of the department and its contribution to 
the continued health and wellbeing of the University. No urgent improvement is necessary, 
although vigilance remains important. 

2 = partially met the Standard for Success 

In the Success Score Rubric, a 2 signifies that attention to the specific area is necessitated in 
order to maintain the ongoing success of the department and its contribution to the continued 
health and wellbeing of the University. This specific area needs to be considered a priority for 
improvement that should be addressed within the next few months. 

1 = did not meet the Standard for Success 


In the Success Score Rubric, a 1 signifies that attention to this area is necessitated immediately 
in order to maintain the ongoing success of the department and its contribution to the 
continued health and wellbeing of the University. This specific area must be considered a high 
priority and addressed immediately. 

X = outcome not addressed; need to revise the outcome or select a different indicator. 

In the Success Score Rubric, an X signifies that the outcome was unable to be measured due 
to a problem with the indicator or data source. A change needs to be made in order to assess 
this outcome. 


Through the use of the standards for success on the Assessment Matrix, the university 
was able to gain a summative evaluation of department and institutional success. This model 
was an effective system to assess both departmental performance and mission fulfillment 
because the subjective nature of the Standard for Success Rubric ensured the space for the 
primary purpose to be achieved: the measurement of change over time. 
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Analysis. The standards for success were established by the department directors, who 
were most knowledgeable in that area and, therefore, most able to apply best practices and 
accepted professional standards to the assessment of their outcomes. However, they were also 
most aware of the minority opinion and the outlier experience. In addition, the standards for 
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success were reviewed by the appropriate dean, supervisor, director, or vice president who was 
also in a strong position to ascertain whether reasonable standards for success were established. 
This step in the process facilitated conversation in regard to departmental expectations and 
served as an opportunity to reconsider the departmental priorities, in addition to the customer 
needs and preferences. 

Thus, standards for success were established and then measured by the directors in 
order to measure improvement in their department over time, and to that end, this system was 
effective. Subsequently, if, over time, a standard for success proved to be set too high or too 
low, it could be adjusted in the same way that outcomes or indicators could be changed. 

Most significantly, this system of assessment was successful in measuring mission 
fulfillment because the core themes were a direct application and representation of the 
mission statement. The core theme measurement ensured that the entirety of the institution 
was represented in the assessment process. In addition, the core themes were designed to be 
formative, as they encouraged the institution to devote human and financial resources to those 
advances most closely tied to the fulfillment of the mission. 

Implementation. In response to changes in the regional accreditation process and 
requirements, the university created an institution-wide assessment process that was initially 
adopted in the fall of 2010 and has been refined through application over the past few years. 
Assessment workshops, in which assessment theory and practice was reviewed, particularly 
related to the categories of the departmental Assessment Matrix, were conducted annually, 
in addition to individual training sessions with department directors. Regular reminders 
as to progression through the assessment timeline were communicated, based on the staff 
Assessment Strategy Cycle (Appendix B) and the academic department Academic Assessment 
Cycle (Appendix C). This assessment framework has received commendations from the 
regional accreditor and recently undergirded a successfully comprehensive evaluation report. 

Findings 

As a result of the last few years of utilizing this assessment process, changes have 
occurred at the departmental, divisional, and institutional levels. At the institutional level, 
the findings from the core themes measurements have resulted in a shift in resources in 
order to address areas of weakness, emphasis and training has occurred in areas that gaps 
were identified, and areas of strength have been communicated in order to celebrate positive 
achievements. The findings from both the core themes measurements and the departmental 
assessment matrices have influenced changes at the divisional level. In some divisions, vice 
presidents integrated the assessment process into the performance evaluation process. In 
others, the assessment process became a tool to facilitate a discussion about departmental 
time usage, workload, and priorities. The public nature of the core themes assessments 
increased the inter-departmental awareness of responsibilities and functions, but also 
resulted in some difficult conversations as unmet goals were published with the assessment 
documents. Finally, at the departmental level the process has proven effective in refining 
the expectations of and priorities for staff department performance and academic student 
learning. In addition, the strong work performed by most departments was obvious and 
documented through the Assessment Matrix; this evidence was used to celebrate the daily 
work of many faculty and staff. 

Culture of Assessment 

Although assessment is standard within academia, the idea of an assessment process 
was unfamiliar to most staff directors. Extensive individual training sessions were necessary 
in order to support the creation of sufficiently broad outcomes, measureable indicators, 
and achievable standards for success. Even among the academic departments, regular 
communication and graduated deadlines were necessary in order to ensure that the deans 
were working with program directors and department chairs throughout the year in a systemic 
and constructive process. A culture of assessment has not yet been created, but most directors 
now appreciate the findings from their matrices and no longer view the Assessment Matrix as 
an additional and unrelated task. 
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Perhaps because of the unfamiliarity many directors had with assessment or perhaps 
simply a symptom that the assessment model is effective, nearly 25% of the original outcomes 
have been modified or deleted, replaced by higher priority department functions and/or 
customer service needs and/or preferences. In addition, many of the indicators have been 
adjusted in order to allow for more accurate, more effective, and/or more contextualized 
measures of success. Finally, the standards for success on the assessment matrices were also 
frequently modified, as surprisingly, most directors initially established unrealistically high 
standards for their measurement (in contrast to establishing a less challenging standard for 
success that could be easily achieved and thus guaranteeing a high score on that outcome). One 
academic dean, reflecting upon the assessment process and model, noted that “the continual 
use, and review, of the assessment matrix has resulted in our college really thinking through our 
program outcomes and determining what is important. In many instances, program outcomes 
have been changed, or eliminated, as a result of authentically using the process.” 


Qualitative Framework 


The focus group setting 
has allowed faculty and 
staff to further elucidate 
important data (especially 
from student narratives 
gathered in the assessment 
process) that could have 
otherwise been considered 
vague or unusable. 


The interpretive approach of the assessment process, in particular the Success Standard 
Rubric which was inherently qualitative in nature, but also allowed for both qualitative and 
quantitative departmental outcome assessment, was necessary in order to achieve the dual 
intentions of the university (assessment of department accountability and improvement, as 
well as mission-fulfillment) and to provide a framework to capture the nuanced and varied 
experiences of stakeholders. However, the very space that allowed this process to successfully 
fulfill the diverse objectives also created frustration for some staff and faculty. Some staff 
questioned the reliability and/or validity of the scores, expressing discomfort with establishing 
their own thresholds for assessment. However, the model proved successful, as over the past 
few years the directors have realized that the standards for success are not intended to be 
consistent across departments, but rather department specific in measuring change over time. 
For example, in one college within the university all faculty and staff meet together in a focus 
group setting to discuss and respond to the scores derived from the assessment process and the 
corresponding data, including narratives from which the scores were obtained. This process 
of providing a venue for faculty and staff to discuss scores with one another (and the change 
in those scores over time), has proven invaluable in the assessment process. Specifically, the 
focus group setting has allowed faculty and staff to further elucidate important data (especially 
from student narratives gathered in the assessment process) that could have otherwise been 
considered vague or unusable. This model does not quantitatively allow for comparison 
between departments or institutions, but rather qualitatively assesses improvement at the 
departmental and institutional levels over time. 


Thus, this applied qualitative framework has proven successful at both the departmental 
and institutional level. The university concluded with a success score of 3.04 on a 4.0 scale for 
the core theme achievement, with scores ranging from a low of 2.71 and a high of 3.4. These 
findings represented an accurate portrayal of the strengths and weaknesses of the university 
and demonstrated a reasonable level and extent of mission fulfillment according to the regional 
accreditors. Further, the university has found that the use of this scale over time has had both 
a motivational and corrective effect on institutional priorities and budgetary allocations. 


Discussion 


The described assessment model was designed to provide a mission-centric focus to 
assessment and to assess all aspects of work across the institution. In addition, it was important 
to create an assessment model that moved away from periodic reporting and to a model 
characterized by ongoing and integrated assessment throughout the academic year (Ewell & 
Jones, 2006). A review of our experience in developing and implementing a new approach to 
institutional assessment was useful in our work to continually refine the process. In addition, 
potentially significant applications also exist for other institutions, especially those that are 
mission focused and open to fundamentally re-thinking their approach to assessment. 
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While quantitative assessment models can be useful for developing metrics which 
can be compared between institutions, these models are insufficient for assessing mission 
fulfillment and departmental efforts within an institution. Furthermore, given the importance 
of linking student learning to institutional mission and culture, assessment approaches must 
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be developed in ways that are consistent with the institutional mission (Contreras-McGavin & 
Kezar, 2007). We found that mission-centric assessment was both fundamental and essentially 
practical in our assessment model. For example, our institution is self-defined as a private 
mission-driven institution. Our educational pedagogy, programs, and student learning 
outcomes are directly linked to the institutional mission. With this in mind, it was necessary 
to develop an approach to assessment that helped us to understand the ways in which we 
were accomplishing our mission and the degree to which we were experiencing success in 
doing so. In addition, a mission-centric approach to assessment was also found to be very 
practical. Specifically, by centering on our mission, focus was given to the process. Faculty, 
staff, administrators, and students were able to better understand both the purpose and goals 
of institutional assessment: to assess how well we do what we say we do. Furthermore, using 
our mixed qualitative and quantitative framework, we were able to learn about the specific 
ways in which constituents experienced mission fulfillment. We found that mission-centric 
assessment was easily understood by those in academic departments, however, it was found to 
be more elusive in non-academic departments. This highlighted our need to continue to build a 
culture of mission-driven work throughout the institution. Understanding mission fulfillment, 
not just in academic programs but across campus, is an essential step in identity formation 
within mission-driven institutions. Mission-centric approaches to assessment become iterative 
in this process; guiding the institution in assessment of mission fulfillment, and then, in turn, 
developing an improved sense of institutional identity across campus. 

Although our previous approach to assessment focused solely on the evaluation of 
academic programs, it was important to develop a new model that involved campus-wide 
participation (Contreras-McGavin & Kezar, 2007). The expansion of assessment across 
campus, while important, was also met with some resistance and confusion. As noted, the 
linkage between academic programs and institutional mission was self-evident. However, 
non-academic departments, for example accounting or maintenance, were unsure of how to 
assess their work within the frame of our institutional mission. This experience highlighted the 
bifurcation that oftentimes occurs within higher education institutions. However, unfortunately, 
this bifurcation has potential negative consequences for stakeholders at all levels. Additionally, 
the qualitative framework and opportunity for qualitative indicator assessment was essential in 
elucidating stakeholders’ varied experiences. For example, students may experience receiving 
one message from within their academic program or from a faculty advisor and another 
message from student accounts or housing - as if the values or principles guiding the work of 
the respective departments are divergent. Engaging in a campus-wide approach to mission¬ 
centric assessment highlighted this bifurcation and the importance of institutions working 
toward creating a consistent and collaborative educational experience. 

In addition to developing an approach to assessment that was both mission-centric, 
and campus-wide, we also purposed to develop a method that was ongoing, integrated and 
iterative (Ewell & Jones, 2006). According to Bresciani (2006), although assessment is typically 
conducted, often the intended results and/or outcomes are not articulated in advance, the 
changes or decisions based on the data and findings are not documented, and the subsequent 
changes are not re-assessed in order to measure whether or not the intended improvements 
were achieved. Similarly, Hanson and Mohn (2011) noted that little attention has been given in 
the literature, to making meaning of collected data and Ewell (2011) highlighted the importance 
of setting expected levels of performance. Our approach to assessment, particularly the 
Assessment Matrix, has aided our institution in engaging in ongoing, and definable, assessment, 
as well as a process of data evaluation that is essentially practical and linked to the change 
process. Regardless of the specific method employed, the inclusion of qualitative analysis is 
essential to ensuring meaningful interpretation and documentation of the collected data. 

Limitations 

Although the interpretive nature of this assessment model has proven both successful 
and necessary in order to assess the diverse functions of the university, the qualitative 
approach limits the ability for inter-departmental comparison. In addition, the mission-centric 
framework also limits the ability for comparison with other institutions. Ultimately, this 
method only measures the department and university improvement over time, benchmarking 
only with ourselves, rather than external indicators. 
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Finally, as discussed previously, a culture of assessment does not yet exist, particularly 
among the staff, at the university. Thus, an ongoing challenge is how to integrate the assessment 
process in a manner that is practical and attainable without adding significant additional work. 
Comprehensive and robust assessment is time consuming, especially within a subjective and 
interpretive model; establishing the importance of this process and the results continues to be 
a challenge for the university. 


Conclusion 

Ultimately, this method Qualitative assessment frameworks are both advantageous and necessary in today’s 

only measures the higher education environments. In response to mandates from both regional and secondary 
department and univer- accreditation bodies, higher education institutions must embrace cultures of assessment, 
sity improvement over Engaging in assessment practices that allow for meaningful comparisons between institutions, 
time, benchmarking only for example, between similar degree programs, can be useful in developing and maintaining 
with ourselves, rather standards or benchmarks for professional fields of practice. I Iowever, in doing so, higher education 
than external indicators, institutions must not lose sight of data that are essentially practical and contextualized within 

the respective institution. Specifically, qualitative assessment frameworks must be utilized in 
order to obtain institutional data that are both contextualized to the institutional environment 
and mission and representative of diverse constituents. Qualitative assessment frameworks 
move beyond the numbers and further elucidate the experience of the stakeholders; the voice 
of diverse students, staff, faculty, and administrators are heard. In this process, meaningful data 
are not only obtained, but a culture of assessment, where participation is valued, is created. 
In this article, the multiple-method assessment model utilized by one small-sized private 
mission-driven institution is outlined. The assessment model, while not solely qualitative in 
nature, is designed to (a) involve stakeholders, at all levels, in the annual assessment process, 
and develop a culture of institutional assessment; (b) develop assessment practices that are 
essentially practical and meaningful for respective departments, resulting in a cycle of ongoing 
assessment and change implementation; and (c) engage in assessment practices that model 
the values and mission of the institution. 
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Appendix A 
Assessment Matrix 


DivisioiV Department: 
FYog-am: 


OUTCOME 

INDICATOR 

DATA 

SOURCE 

TIME 

PBIOD 

STANDARD 

FOR 

SUOCHB 

FINDINGS 

ANALY9S 

OF 

HFUNGS 

SUCCSS 

SCOPE 

RESULTING 

fflOGRAM 

CHANGES 

00RET1CME 

ADOFE98ED 






















Assessment Key: 

OUTCOME: The department or program outcomes 

INDICATOR: W hat will be measured that directly or indirectly indicates mastery 
of the outcome? 

DATA SOURCE: What student action or product is being evaluated? 

TIME PERIOD: When will the data source information be collected? 

STANDARD FOR SUCCESS: What level of accomplishment will be considered satisfactory for meeting the 
outcome goal? 

FINDINGS: What level of accomplishment was observed? 

ANALYSIS OF FINDINGS: What is the conclusion regarding meeting the outcome? 

SUCCESS SCORE: 

X = outcome not addressed; need to revise the outcome or select different indicator. 

1 = did not meet Standard for Success 

2 = partially met the Standard for Success 

3 = met the Standard for Success 

4 = significantly exceeded the Standard for Success 

RESULTING PROGRAM CHANGES: What changes in the program will be made to improve the success score? 
CORE THEME ADDRESSED: What Institutional Core Theme is related to this outcome? 

Core Theme One: Building a caring community and enduring culture 
Core Theme Two: Developing Christian commitment and Spirit-formed lives 

Core Theme Three: Advancing academic engagement through teaching, learning and scholarly production 

Core Theme Four: Empowering people with the vision and tools to meet human need in their personal and 
professional lives 
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Assessment Matrix: Department (Staff) Example 


OUTCOME 

INDICATOR 

DATA 

90UFCE 

TIME 

PffflOD 

STANDARD 

FOR 

SUCCESS 

RNDINGS 

ANALY9S 

OF 

RNDINGS 

SUOCESS 

900FE 

RESULTING 

PROGRAM 

CHANGES 

CORE 

THBV1E 

ADDRESS) 

9udentsare 
prepared for 
success when 
entering the 

LEAP or Ministry 
Head 9 art 

programs 

Orientation 

Attendance at 

orientation 

Beginning of 
Program 

4-80% of new 

students 

attend 

3-75% 

2-70% 

1 -65% 

April 2012 
Orientation: 

86% attendance 
(37/43 eligible 
students) 

Importance of 
attendance at 

Orientation was 

communicated to 
students by 
Enrollment, Off 
staff, and 
advisors. 

4 

None 

1 

9udentsknow 

who their 

advisor is and 

understand their 
academic plan to 
graduate. 

Advising 
appointments 
and academic 
plan 

Advising 
schedule and 
transfer reports 

Beginning of 
each semester 

4-80% of 

students meet 

with advisor 
prior to the 
semester to 

review their 
academic plan 
3-75% 

2-70% 

1-65% 

April 2012 
semester start: 

135 of 175 
students (77%) 
were advised at 

least one month 
prior to the 
beginning of 
semester. 

9udentswere 
contacted by 
off® 3 sf f and 

advisors and 
encouraged to 
complete the 
advising process 
in advance. 

3 

Ongoing 

conversation 

about the best 
dates to open 
registration and 
for facilitating 
advising 
appointments 

1 

9udents 
currently 
enrolled are 

satisfffiJ wth 

their educational 
experience. 

Ftesponseson 
annual survey 

Adult 9udent 
Priorities 9jrvey 
question: “Ftete 
your overall 
satisfaction 
with your 
experience here 
thusfar.” 

Spring 

semester 

4-80% of 

students 

indicate 

satisfffil or 
higher 

3-75% 

2-70% 

1-65% 

Spring 2012 
survey result: 
5.69 mean for 

all LB\P 

students. 

Average response 
was between 

somewhat 

satisfffil and 

satisfffil. 

2 

9andard needs 

to be rewritten 
to accurately 
reftet resil ts 
provided from 
survey. 

1 

9udents 
complete the 
LEAP program 
with afeeling 
that they have 
made a 

worthwhile 

investment. 

Ftesponseson 
exit survey 

Exit survey 
question: 
“Tuition paid is 
a worthwhile 

investment” 

Cbmpletion of 
program 

4-80% of 
graduates 
mark sat isfsd 
or higher 
3-75% 

2-70% 

1-65% 

April 2012 Bat 
9jrveys: 65% 

Psychology: 6 of 

11 

Ministry 
Leadership: 6 of 

10 

Business 
Management: 12 
of 16 

1 

9andard for 

success needs to 

be rewritten for 

a more realistic 
approach. 

4 

New adjunct 
instructors are 
effectively 
trained before 
teaching in Off 
programs 

Orientation for 
faculty 

Meeting 
schedule with 
faculty 

Meeting with 
faculty 3 
months prior 
to their ffst 

course 

4-80% of 
faculty meet 
with Off staff 

3 months prior 
to fist (curse 

3-75% 

2-70% 

1-65% 

4 of 5 new 
professors 
starting 
between 

January 1 and 
April 30 
completed the 
Off orientation 

process. 

The one 

instructor who 
did not complete 
an orientation is a 
residential faculty 
member. Follow 
up was conducted 
during and after 
the course. 

4 

Off personnel 
are considering 
how best to 

continue 

orientation with 
faculty through 
their fist (curse 

to ensure the 
best experience. 

3 

Current Off 
faculty have the 
opportunity for 
professional 
development. 

Development 
workshops and 
online 

development 

courses 

Attendance at 
workshops and 
use of online 

courses 

Minimum of 1 
workshop 
scheduled for 

each semester 

4-20 or more 
faculty in 
attendance 

3-15 

2-12 

1-10 

Two spring 
workshops were 
conducted with 

attendance of 

30 and 15. 

Attendance 

declined for the 
second workshop 
but positive 
feedback was 

received. 

3 

A comprehensive 
schedule for 
workshops and 
topics for the 
2012-13 
academic year 
will be 
developed in 
summer 2012. 

3 

Curriculum and 
delivery models 
consistently 
evaluated for 
possible 
improvement. 

Ftesponseson 

course 

evaluations 

Cburse 

evaluations 

Conclusion of 

each class 

4-80% of 

course 

evaluations 

score 4.0 or 
higher 

3-75% 

2-70% 

1-65% 

Course 

evaluations 
processed from 
January 1 
through April 

30:45 of 48 
classes (94°/c) 
rated 4.0 or 
higher 

For the most part, 
positive feedback 
received from 

students 
regarding 
curriculum and 
delivery models. 

4 

9andard for 

success may 

need to be 
adjusted to a 
higher 
expectation. 

3 
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Assessment Matrix: Program (Academic) Example 


OUTCOME 

INDICATOR 

DATA 

90URCE 

TIME 

PffaOD 

STANDARD 

FOR 

SUCCESS 

HNDINGS 

ANALYSIS 

OF 

RNDINGS 

SUCCESS 

SOORE 

RESULTING 

PROGRAM 

CHANGES 

NU 

CORE 

THHVIE 

Traditional Undergraduate - Bachelors of Arts in Psychology 

Demonstrate 
competency in 
academic 
content areas 
that meet 
prerequisites 
for graduate 
work in 
behavioral and 
social sciences; 

9udent 
performance 
on ETSMajor 
Reid Test in 
Psychology. 

Psyc4403 
Advanced 
Topics in 
Psychology - 
ETSMajor 

Reid Test in 
Psychology 

Data is 
reported by 
the 

Advanced 
Topics 
professor to 
the 

undergrad 
psychology 
chair at the 
end of 

Spring 

semester. 

The mean 
score of NU 
undergrad 
psychology 
students is 
above the 50 th 
percentile 
thereby 
demonstrating 
competency in 
academic 
content areas 
of behavioral 
and social 
sciences 

• 1 <;40% 

• 2=41%- 
49% 

• 3 = 50%- 
55% 

• 4 >56% 

Due to an 
unforeseen 
circumstance 
the ETS 

Major Reids 
Test was not 
administered 
to students. 

Suggestion: 

CSS 

develops its 
own 

Cdpstone 

Bam to 
administer to 
student. 

X 

CSS 

undergraduate 
faculty will 
develop an NU 
psychology 
exam to 

administer in the 
2014-2015 
academic year. 
Rationale: By 
developing our 
own test, CSS 
faculty will be 
better able to 
determine which 
content areas 
students are the 
strongest and 
weakest in. 
However, for the 
2013-2014 
academic year, 
faculty will 
continue to use 
the ETSMajor 

Reid Test. 

3 

Demonstrate 

theabilitvto 

effectively 

communicate 

their 

understanding 

of 

osvcholocjcal 

issues in both 

oral and 

written 
formats 
including 
mastery of APA 
style 

Student 
performance 
on 4 key 
elements of 
oral commun¬ 
ication on a 

Qass 

Presentation 

Assignment 

Psyc4203 
Cbgnitive 
Psychology- 
Qass 

Resent at ion 
Assignment 

Data is 
reported by 
the 

Cbgnitive 
Psychology 
professor to 
the 

undergrad 
psychology 
chair at the 
end of Fall 
semester. 

50% of 
students 
demonstrate 
an ability to 
effectively 
communicate 
orally by 
achieving 80% 
(B-) or higher 
on the 4 key 
elements of 
oral commun¬ 
ication. 

• 1 =s39% 

• 2=40%- 
49% 

• 3 = 50%- 
55% 

• 4 >56% 

97% of 
students 
scored 80% 
or higher 
on thefour 
key 

elements of 
oral 

commun¬ 

ication. 

Only one 
students did 
not score 

80% percent 
or higher on 
key element 
“Quality of 
Presentation” 

4 

Ftesulting 

Change: All 
standards will be 
revised by 
f acuity for 2013- 
2014. The goal 
of the revision is 
to standardize 
the outcome 
standards within 
each program. 

3 

Demonstrate 
the ability to 
effectively 
communicate 
their 

understanding 

of 

psychological 
issues in both 
oral and 

written 

formats 
indudinq 
mastery of 

APA stvle: 

Student 
performance 
on 6 key 
sections of the 
Rnal Research 
Paper. 

Psyc 3023 
Research 
Methods II- 
Rnal 

Research 

Paper 

Data is 
reported by 
the 

Research 
Methods II 
professor to 
the 

undergrad 
psychology 
chair at the 
end of the 
Spring 
semester. 

50% of 
students 
demonstrate 
an ability to 
communicate 
in writing with 
a mastery of 

APA style by 
achieving 80% 
(B-) or higher 
on the 6 key 
sections of the 
Rnal Ftesearch 
F&per. 

• 1 <39% 

• 2=40%- 
49% 

• 3 = 50%- 
55 % 

• 4 ^ 56% 

95% of 
students 
demonstrate 
dan ability 
to 

communicat 
e in written 
format 
including 
mastery of 
APA style. 

Nothing 

noted 

4 

Ftesulting 

Change: All 
standards will be 
revised by 
f acuity for 2013- 
2014. The goal 
of the revision is 
to standardize 
the outcome 
standards within 
each program. 

3 
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Assessment Matrix: Program (Academic) Example 


OUTCOME 

INDICATOR 

DATA 

SOURCE 

TIME 

PffflOD 

STANDARD 

FOR 

SUCCESS 

HNDINGS 

ANALYSIS 

OF 

RNDINGS 

SUCCESS 

SOORE 

RESULTING 

PROGRAM 

CHANGES 

NU 

CORE 

THEME 

Traditional Underg-aduate - Bachelors of Artsin Psychology 

Think critically 
concerning 
issues relating 
to psychology, 
culture, and 
social injustice 
within the 
context of a 
globalized 
world; 

Student 
ability to 
think critically 
concerning 
issues of 
psychology, 
culture, and 
social 

injustice as 
demonstrated 
in 3 sections 
of a ref tet i ve 
writing 
assignment 

Rsyc 4973 
International 
Reid Study- 
Student 
Journals 

Data is 
reported by 
the 

International 
Reid Study 
professor to 
the 

undergrad 
psychology 
chair at the 
end of the 
faring 
Semester. 

80% of 
students 
demonstrate 
an ability to 
think critically 
in areas of 
psychology, 
culture, and 
social injustice 
by achieving 
an 80% or 
higher on the 

3 sections of a 
reftetive 
writing 
assignment. 

• 1 <69% 

• 2=70-79% 

• 3 = 80- 
89% 

• 4 ^ 90% 

77% of 
students 
demonst¬ 
rated an 
ability to 
think 

critically in 
areas of 
psychology 
culture, and 
social 

injustice by 
achieving an 
80% or 
higher on a 
ref tet i \e 
writing 
assignment 

Several 

submitted 

the 

assignment 
late and, 
therefore, 
lost points; 
this affected 
their overall 
grade on the 
assignment. 

2 

Ftesulting 

Change: All 
standards will 
be revised by 
faculty for 
2013-2014. The 
goal of the 
revision is to 
standardize the 
outcome 
standards 
within each 
program. 

3 

Examine 
Christian faith 
integration 
and 

psychology. 

Sudent 
integration of 
psychology 
and Christian 
faith as 
demonstrated 
on a paper 
about a 
fit i anal 
meeting of 
Sgmund 

Freud and CS 
Lewis. 

FSyc4403 
Advanced 
Topics in 
Psychology 

Data is 
reported by 
the Advanced 
Topics in 
Psychology 
professor to 
the 

undergrad 
psychology 
chair at the 
end of the 
semester. 

80% of 
students 
successfully 
integrate 
psychology 
and Christian 
faith by 
achieving 80% 
or higher on a 
paper about a 
meeting 
between 

Freud and 

Lewis. 

• 1 <69% 

• 2=70-79% 

• 3 = 80- 
89% 

• 4 >90% 

100% of 

students 

successfully 

integrated 

psychology 

and 

Christian 

faith. 

Nothing 

noted 

4 

Ftesulting 

Change: All 
standards will 
be revised by 
faculty for 
2013-2014. The 
goal of the 
revision is to 
standardize the 
outcome 
standards 
within each 
program. 

2 
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Appendix B 

Assessment Strategy Cycle 
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Appendix G 

Academic Assessment Cycle 
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Fall: 

Sbmester of 
Decisions 


Gathering of 
Assessment Data 

Collect the Findings Data for 
the Outcomes and Indicators 
throughout the year 


Curriculum changes are 
considered by Academic 
Affairs Committee and 
Faculty Council 


Annual Deans Assessment 

1 1 
; Board 1 

Report to Provost 

1 Meeting 1 


Deans suggest revisions to 
individual program 
curriculum 


Student Learning/ 
Customer Service Outcomes 

Establish Outcomes, Indicators, 
Data source, Time Period, and 
Standards for Success 
Assign a Core Theme for each 
Outcome 




Complete Analysis of 
Findings based on 
Outcomes, Indicators, 
and Findings Data 
Assign a Success Score 
for each Outcome 


Resulting Program Changes 

Based on analysis, plan subsequent Program 
Changes 

Possibly make adjustment to future 
Outcomes, Indicators, and/or Standards for 
Success 

Implement Program Changes for next 
assessment cycle 

Send completed Assessment Matrix to 
Supervisor and Assistant Provost 
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